Cross-Modal Effects on Perception 1 Seeing Sound: Changing Visual Perception Through Cross-Modal Interaction

A single flash of light accompanied by a series of beeps gives the impression of flickering (Shams, Kamitani, & Shimojo, 2000). Thus, an unambiguous visual stimulus appears different when paired with an auditory stimulus. Similarly, with a flashing pattern, we find that beeps can increase or decrease perceived number of flashes. Furthermore, the perceived number or flashes can either improve or impair objectively measured ability to judge the pattern’s orientation. Extra sound-induced flashes facilitate performance as much as an additional patch of pattern. Although much prior research has asserted visual dominance in cross-modal interaction, our results indicate that in certain tasks the auditory sense can alter visual perception and performance. Cross-Modal Effects on Perception 3 As we move through the world we are continually taking in information what we see, feel, hear, etc. and relating this new information to previous memories. Sometimes the information received by the various sensory modalities can be in conflict. For example, you may see a ventriloquist’s dummy’s lips moving, but hear the sound coming from the ventriloquist. However, according to your visual system, the sound seems to be coming from the dummy’s moving mouth. According to your auditory system, the sound seems to be coming from the ventriloquist’s mouth. Your perceptual system is faced with the challenge of incorporating these inconsistent signals into a coherent impression. Partly because the senses have mostly been studied independently of one another, the way that integration of information across modalities occurs in situations such as this one is not well understood. “The illusory flash effect” is based on the integration of the visual and auditory senses (Shams, Kamitani, & Shimojo, 2000). A disc is flashed once and is accompanied by one to three beeps. When the flashed disc is accompanied by more than one beep, it appears to flash twice. The extra perceived flash is illusory (Shams et al., 2000). We wondered whether the extra perceived flash is equivalent to an additional visual event. Does the extra “event” contribute to how well we see, as measured by objective performance? In order to address these questions, two tasks were developed for this study based on a modified version of the original illusion. A flashing gabor patch was used in the place of the original flashing white disc. The modified illusion is called flicker flutter. Cross-Modal Effects on Perception 4 Other cross-modal research. Other illusions demonstrate that auditory stimuli can affect the interpretation of an ambiguous visual situation. One example is an illusion in which two circles appear on opposite sides of a display. The circles move smoothly toward one another, meeting in the middle. At the midpoint, the circles’ paths are unclear and they appear to either pass through one another or bounce off each other. (Metzger, 1934). The display is visually ambiguous in that either interpretation is supported by the visual stimuli. If a beep or click is played at the time that the circles appear to pass through one another, the circles are interpreted as bouncing off one another (Sekuler, Sekuler, & Lau, 1997). The sound disambiguates an ambiguous display. By comparison, the visual display in the Shams et al. (2000) illusion is unambiguous. Each presentation contains only one flash. However, when more than one beep is played, the observer sees two flashes. Instead of causing the observer to choose between two different, equally valid interpretations of the display, the sound actually causes the observer to see something that is not there. Visual dominance. Many studies support the concept of visual dominance in cross-modal interaction. In other words, if a person receives conflicting information from two different sensory modalities, and one of the modalities is vision, then the visual input will dominate. A classic example is the McGurk Effect, discovered by McGurk and MacDonald (1976). Observers seeing a woman mouthing the syllable “va,” and hearing an auditory recording of a woman pronouncing “ba,” report hearing “va.” The inconsistent auditory stimulus is overridden by the visual stimulus which induces a consistent auditory experience. Other trials with different syllables resulted in reported syllables that combined the visual and auditory input. Cross-Modal Effects on Perception 5 Other cross-modal research has demonstrated that visual adaptation can produce auditory aftereffects. One such experiment explored how adaptation to a moving visual stimulus could affect the perceived location of a subsequently presented auditory stimulus, and how the reverse situation affected the perceived location for a visual stimulus (Kitagawa & Ichihara, 2002). Kitigawa & Ichihara (2002) point to the fact that visual adaptation was more likely to affect auditory perception than vice versa as proof that the visual system is more precise and influential in terms of spatial perception. One explanation of this result is that the internal representations (spatial) that were best suited to decision making were visual, so the visual system dominated. One of the reasons that the illusory flash effect is so intriguing is that it contradicts the usual finding of visual dominance in cross-modal interaction (Shams et al., 2000). Despite unambiguous visual information, the auditory stimulus changes the visual experience rather than vice versa. It is due to this unexpected result that further exploration of this illusion seems necessary. Through a deeper understanding of this illusion, perhaps we can develop a better grasp of how cross-modal integration works. We used a subjective matching task to measure how many apparent flashes could be produced by the flicker flutter illusion. This task was used to identify how different conditions affected subjective perception of flashes. By knowing exactly how many illusory flashes were being perceived in different conditions, it was possible to explore how performance in an objective task was affected by different numbers of temporal objects (illusory or actual flashes) and different numbers of spatial objects (gabor patches). By comparing these effects we could answer the first question of whether extra perceived flashes are equivalent to additional events. We used an objective orientation Cross-Modal Effects on Perception 6 discrimination task to address this question as well as the second question of whether this change in perceived events actually affects how well you see. Matching Task